adding language analyzers #8591

AntonEliatra · 2024-10-21T15:08:57Z

Description

adding arabic language analyzer

Issues Resolved

Part of #1483 addressed in this PR.

Version

all

Checklist

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license and subject to the Developers Certificate of Origin.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

Signed-off-by: Anton Rubin <[email protected]>

github-actions · 2024-10-21T15:09:10Z

Thank you for submitting your PR. The PR states are In progress (or Draft) -> Tech review -> Doc review -> Editorial review -> Merged.

Before you submit your PR for doc review, make sure the content is technically accurate. If you need help finding a tech reviewer, tag a maintainer.

When you're ready for doc review, tag the assignee of this PR. The doc reviewer may push edits to the PR directly or leave comments and editorial suggestions for you to address (let us know in a comment if you have a preference). The doc reviewer will arrange for an editorial review.

Signed-off-by: Fanit Kolchina <[email protected]>

Signed-off-by: Anton Rubin <[email protected]>

…n analyzer docs Signed-off-by: Anton Rubin <[email protected]>

…lithuanian,norwegian and persion laguage analyzer docs Signed-off-by: Anton Rubin <[email protected]>

…rkish language analyzer docs Signed-off-by: Anton Rubin <[email protected]>

kolchfa-aws

Thank you, @AntonEliatra! Please apply my changes, and we'll get this to editorial review.

_analyzers/language-analyzers/index.md

kolchfa-aws · 2024-11-06T16:19:22Z

_analyzers/language-analyzers/index.md

+nav_order: 100
+parent: Analyzers
+has_children: true
+has_toc: false


I think it's not a bad idea to have the TOC here. It will list all analyzers on the bottom under Related articles.

kolchfa-aws · 2024-11-06T17:38:30Z

_analyzers/language-analyzers/index.md

+
+#### Example request
+
+The following query specifies the `french` language analyzer for the index `my-index`:


This request creates a text field with a french subfield configured with the french analyzer. I think we can keep this request but correct the description for it. Also, it would be nice to provide just a plain example with setting an analyzer on the whole index.

I added additional example

_analyzers/language-analyzers/index.md

kolchfa-aws · 2024-11-06T17:55:24Z

_analyzers/language-analyzers/spanish.md

+
+## Stem exclusion
+
+You can also use `stem_exclusion` with this language analyzer using the following command:


Suggested change

You can also use `stem_exclusion` with this language analyzer using the following command:

You can use `stem_exclusion` with this language analyzer using the following command:

kolchfa-aws · 2024-11-06T17:55:33Z

_analyzers/language-analyzers/swedish.md

+
+## Stem exclusion
+
+You can also use `stem_exclusion` with this language analyzer using the following command:


Suggested change

You can also use `stem_exclusion` with this language analyzer using the following command:

You can use `stem_exclusion` with this language analyzer using the following command:

kolchfa-aws · 2024-11-06T17:55:42Z

_analyzers/language-analyzers/thai.md

+
+## Stem exclusion
+
+You can also use `stem_exclusion` with this language analyzer using the following command:


Suggested change

You can also use `stem_exclusion` with this language analyzer using the following command:

You can use `stem_exclusion` with this language analyzer using the following command:

kolchfa-aws · 2024-11-06T17:55:51Z

_analyzers/language-analyzers/turkish.md

+
+## Stem exclusion
+
+You can also use `stem_exclusion` with this language analyzer using the following command:


Suggested change

You can also use `stem_exclusion` with this language analyzer using the following command:

You can use `stem_exclusion` with this language analyzer using the following command:

kolchfa-aws · 2024-11-06T17:57:41Z

_analyzers/language-analyzers/arabic.md

+
+## Arabic analyzer internals
+
+The `arabic` analyzer is build using the following:


Global: please apply these changes to all files.

Co-authored-by: kolchfa-aws <[email protected]> Signed-off-by: AntonEliatra <[email protected]>

Signed-off-by: Anton Rubin <[email protected]>

Signed-off-by: AntonEliatra <[email protected]>

Signed-off-by: Anton Rubin <[email protected]>

kolchfa-aws

LGTM

natebower

@kolchfa-aws @AntonEliatra Please see my comments and changes and let me know if you have any questions. Thanks!

natebower · 2024-11-11T15:50:57Z

_analyzers/language-analyzers/armenian.md

+
+## Custom Armenian analyzer
+
+You can create custom Armenian analyzer using the following command:


Suggested change

You can create custom Armenian analyzer using the following command:

You can create a custom Armenian analyzer using the following command:

natebower · 2024-11-11T15:51:55Z

_analyzers/language-analyzers/basque.md

+
+## Custom Basque analyzer
+
+You can create custom Basque analyzer using the following command:


Suggested change

You can create custom Basque analyzer using the following command:

You can create a custom Basque analyzer using the following command:

natebower · 2024-11-11T15:52:33Z

_analyzers/language-analyzers/bengali.md

+
+## Custom Bengali analyzer
+
+You can create custom Bengali analyzer using the following command:


Suggested change

You can create custom Bengali analyzer using the following command:

You can create a custom Bengali analyzer using the following command:

natebower · 2024-11-11T15:53:26Z

_analyzers/language-analyzers/brazilian.md

+
+## Custom Brazilian analyzer
+
+You can create custom Brazilian analyzer using the following command:


Suggested change

You can create custom Brazilian analyzer using the following command:

You can create a custom Brazilian analyzer using the following command:

natebower · 2024-11-11T15:56:56Z

_analyzers/language-analyzers/bulgarian.md

+
+## Custom Bulgarian analyzer
+
+You can create custom Bulgarian analyzer using the following command:


Suggested change

You can create custom Bulgarian analyzer using the following command:

You can create a custom Bulgarian analyzer using the following command:

natebower · 2024-11-11T16:37:57Z

_analyzers/language-analyzers/thai.md

+
+- Tokenizer: `thai`
+
+- Token Filters:


Suggested change

- Token Filters:

- Token filters:

natebower · 2024-11-11T16:38:08Z

_analyzers/language-analyzers/thai.md

+
+## Custom Thai analyzer
+
+You can create custom Thai analyzer using the following command:


Suggested change

You can create custom Thai analyzer using the following command:

You can create a custom Thai analyzer using the following command:

natebower · 2024-11-11T16:38:46Z

_analyzers/language-analyzers/turkish.md

+
+- Tokenizer: `standard`
+
+- Token Filters:


Suggested change

- Token Filters:

- Token filters:

natebower · 2024-11-11T16:38:55Z

_analyzers/language-analyzers/turkish.md

+
+## Custom Turkish analyzer
+
+You can create custom Turkish analyzer using the following command:


Suggested change

You can create custom Turkish analyzer using the following command:

You can create a custom Turkish analyzer using the following command:

_analyzers/language-analyzers/index.md

Co-authored-by: Nathan Bower <[email protected]> Signed-off-by: AntonEliatra <[email protected]>

Co-authored-by: Nathan Bower <[email protected]> Signed-off-by: kolchfa-aws <[email protected]>

Signed-off-by: Fanit Kolchina <[email protected]>

* adding arabic language analyzer Signed-off-by: Anton Rubin <[email protected]> * Add grandparent to arabic analyzer Signed-off-by: Fanit Kolchina <[email protected]> * adding more details Signed-off-by: Anton Rubin <[email protected]> * adding armenian language analyzer Signed-off-by: Anton Rubin <[email protected]> * adding basque bengali and brazilian language analyzers Signed-off-by: Anton Rubin <[email protected]> * adding bulgarian catalan and cjk language analyzers Signed-off-by: Anton Rubin <[email protected]> * adding czech,danish,dutch,english,estonian,finnish,french and galician analyzer docs Signed-off-by: Anton Rubin <[email protected]> * adding german,greek,hindi,hungarian,indonesian,irish,italian,latvian,lithuanian,norwegian and persion laguage analyzer docs Signed-off-by: Anton Rubin <[email protected]> * adding portuguese,romanian,russian,sorani,spanish,swedish,thai and turkish language analyzer docs Signed-off-by: Anton Rubin <[email protected]> * Apply suggestions from code review Co-authored-by: kolchfa-aws <[email protected]> Signed-off-by: AntonEliatra <[email protected]> * updating as per pr review Signed-off-by: Anton Rubin <[email protected]> * fixing broken link Signed-off-by: Anton Rubin <[email protected]> * Apply suggestions from code review Co-authored-by: Nathan Bower <[email protected]> Signed-off-by: AntonEliatra <[email protected]> * Update _analyzers/language-analyzers/index.md Co-authored-by: Nathan Bower <[email protected]> Signed-off-by: kolchfa-aws <[email protected]> * Add redirect to index page Signed-off-by: Fanit Kolchina <[email protected]> --------- Signed-off-by: Anton Rubin <[email protected]> Signed-off-by: Fanit Kolchina <[email protected]> Signed-off-by: AntonEliatra <[email protected]> Signed-off-by: kolchfa-aws <[email protected]> Co-authored-by: Fanit Kolchina <[email protected]> Co-authored-by: kolchfa-aws <[email protected]> Co-authored-by: Nathan Bower <[email protected]> (cherry picked from commit c29761c) Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

adding arabic language analyzer

258668b

Signed-off-by: Anton Rubin <[email protected]>

github-actions bot assigned kolchfa-aws Oct 21, 2024

kolchfa-aws assigned vagimeli and unassigned kolchfa-aws Oct 21, 2024

kolchfa-aws and others added 8 commits October 21, 2024 11:26

Add grandparent to arabic analyzer

37b845a

Signed-off-by: Fanit Kolchina <[email protected]>

adding more details

436bd3e

Signed-off-by: Anton Rubin <[email protected]>

adding armenian language analyzer

93308e4

Signed-off-by: Anton Rubin <[email protected]>

adding basque bengali and brazilian language analyzers

d416dd4

Signed-off-by: Anton Rubin <[email protected]>

adding bulgarian catalan and cjk language analyzers

2e4f01d

Signed-off-by: Anton Rubin <[email protected]>

adding czech,danish,dutch,english,estonian,finnish,french and galicia…

8a1052d

…n analyzer docs Signed-off-by: Anton Rubin <[email protected]>

adding german,greek,hindi,hungarian,indonesian,irish,italian,latvian,…

15c0f8c

…lithuanian,norwegian and persion laguage analyzer docs Signed-off-by: Anton Rubin <[email protected]>

adding portuguese,romanian,russian,sorani,spanish,swedish,thai and tu…

2b2845f

…rkish language analyzer docs Signed-off-by: Anton Rubin <[email protected]>

AntonEliatra marked this pull request as ready for review November 4, 2024 12:26

AntonEliatra requested review from kolchfa-aws, Naarcha-AWS, vagimeli, AMoo-Miki, natebower, dlvenable and epugh as code owners November 4, 2024 12:26

AntonEliatra changed the title ~~adding arabic language analyzer~~ adding language analyzers Nov 4, 2024

vagimeli assigned kolchfa-aws Nov 4, 2024

vagimeli added 3 - Tech review PR: Tech review in progress 4 - Doc review PR: Doc review in progress Content gap labels Nov 4, 2024

kolchfa-aws requested changes Nov 6, 2024

View reviewed changes

kolchfa-aws added the backport 2.18 PR: Backport label for 2.18 label Nov 6, 2024

kolchfa-aws unassigned vagimeli Nov 6, 2024

Apply suggestions from code review

3fc50b1

Co-authored-by: kolchfa-aws <[email protected]> Signed-off-by: AntonEliatra <[email protected]>

AntonEliatra added 3 commits November 7, 2024 12:34

updating as per pr review

26fbb9b

Signed-off-by: Anton Rubin <[email protected]>

Merge branch 'main' into adding-language-analyzers-docs

310f734

Signed-off-by: AntonEliatra <[email protected]>

fixing broken link

5172a0d

Signed-off-by: Anton Rubin <[email protected]>

kolchfa-aws approved these changes Nov 11, 2024

View reviewed changes

natebower reviewed Nov 11, 2024

View reviewed changes

kolchfa-aws and others added 5 commits November 11, 2024 15:20

Merge branch 'main' into adding-language-analyzers-docs

6d2089b

Apply suggestions from code review

719ef66

Co-authored-by: Nathan Bower <[email protected]> Signed-off-by: AntonEliatra <[email protected]>

Update _analyzers/language-analyzers/index.md

e29f690

Co-authored-by: Nathan Bower <[email protected]> Signed-off-by: kolchfa-aws <[email protected]>

Merge branch 'main' into adding-language-analyzers-docs

25e6771

Add redirect to index page

3a429eb

Signed-off-by: Fanit Kolchina <[email protected]>

kolchfa-aws merged commit c29761c into opensearch-project:main Nov 14, 2024
5 checks passed

opensearch-trigger-bot bot mentioned this pull request Nov 14, 2024

[Backport 2.18] adding language analyzers #8757

Merged

github-actions bot pushed a commit that referenced this pull request Nov 14, 2024

adding language analyzers (#8591) (#8757)

03360a6

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

adding language analyzers #8591

adding language analyzers #8591

AntonEliatra commented Oct 21, 2024

github-actions bot commented Oct 21, 2024

kolchfa-aws left a comment

kolchfa-aws Nov 6, 2024

AntonEliatra Nov 7, 2024

kolchfa-aws Nov 6, 2024

AntonEliatra Nov 7, 2024

kolchfa-aws Nov 6, 2024

kolchfa-aws Nov 6, 2024

kolchfa-aws Nov 6, 2024

kolchfa-aws Nov 6, 2024

kolchfa-aws Nov 6, 2024

kolchfa-aws left a comment

natebower left a comment

natebower Nov 11, 2024

natebower Nov 11, 2024

natebower Nov 11, 2024

natebower Nov 11, 2024

natebower Nov 11, 2024

natebower Nov 11, 2024

natebower Nov 11, 2024

natebower Nov 11, 2024

natebower Nov 11, 2024


		#### Example request

		The following query specifies the `french` language analyzer for the index `my-index`:


		## Stem exclusion

		You can also use `stem_exclusion` with this language analyzer using the following command:

	You can also use `stem_exclusion` with this language analyzer using the following command:
	You can use `stem_exclusion` with this language analyzer using the following command:


		## Arabic analyzer internals

		The `arabic` analyzer is build using the following:


		## Custom Armenian analyzer

		You can create custom Armenian analyzer using the following command:

	You can create custom Armenian analyzer using the following command:
	You can create a custom Armenian analyzer using the following command:


		## Custom Basque analyzer

		You can create custom Basque analyzer using the following command:

	You can create custom Basque analyzer using the following command:
	You can create a custom Basque analyzer using the following command:


		## Custom Bengali analyzer

		You can create custom Bengali analyzer using the following command:

	You can create custom Bengali analyzer using the following command:
	You can create a custom Bengali analyzer using the following command:


		## Custom Brazilian analyzer

		You can create custom Brazilian analyzer using the following command:

	You can create custom Brazilian analyzer using the following command:
	You can create a custom Brazilian analyzer using the following command:


		## Custom Bulgarian analyzer

		You can create custom Bulgarian analyzer using the following command:

	You can create custom Bulgarian analyzer using the following command:
	You can create a custom Bulgarian analyzer using the following command:


		## Custom Thai analyzer

		You can create custom Thai analyzer using the following command:

	You can create custom Thai analyzer using the following command:
	You can create a custom Thai analyzer using the following command:


		## Custom Turkish analyzer

		You can create custom Turkish analyzer using the following command:

	You can create custom Turkish analyzer using the following command:
	You can create a custom Turkish analyzer using the following command:

adding language analyzers #8591

adding language analyzers #8591

Conversation

AntonEliatra commented Oct 21, 2024

Description

Issues Resolved

Version

Checklist

github-actions bot commented Oct 21, 2024

kolchfa-aws left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kolchfa-aws left a comment

Choose a reason for hiding this comment

natebower left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment